Fix: MoE Memory Budgeting and Dynamic Multimodal Tokens - #98
Conversation
|
Thanks @roydsouza — and credit where due: the multimodal BOA/EOA half of this landed via your #104 in May, and it has been on main with tests since ( What remains unique here is the MoE memory buffer and the swap-usage warning, and both are worth having. One fix needed on the swap monitor: the hand-declared This branch also carries an accidental working-tree state that reverts ~2,300 lines of unrelated shipped code (speculative-decoding dispatch, ModelDownloader, SettingsView, several safety gates), so rather than rebasing it, a fresh small PR on current main with just those ~40 lines — the Closing in favour of the pieces that already landed (#104, #114). The buffer + swap-warning invitation stands. |
Description
This PR addresses two critical issues identified during an adversarial audit of SwiftLM on Apple Silicon (M5):
boaToken(255010) andeoaToken(255011) by extracting them dynamically fromconfig.json. This fixes expert routing for non-Qwen multimodal models.Changes
moeBuffertocomputeSSDMemoryBudget.sysctl vm.swapusagecheck inServer.swift.extractMultimodalTokensinOmniModelFactoryto resolve BOA/EOA tokens from config.Verified on Apple Silicon M5 with Gemma 4 MoE.